Bayesian Model Averaging in Rule Induction

Bayesian model averaging (BMA) can be seen as the optimal approach to any induction task. It can reduce error by accounting for model uncertainty in a principled way, and its usefulness in several areas has been empirically veri ed. However, few attempts to apply it to rule induction have been made. This paper reports a series of experiments designed to test the utility of BMA in this eld. BMA is applied to combining multiple rule sets learned from di erent subsets of the training data, to combining multiple rules covering a test example, to inducing technical rules for foreign exchange trading, and to inducing conjunctive concepts. In the rst two cases, BMA is observed to produce lower accuracies than the ad hoc methods it is compared with. In the last two cases, BMA is observed to typically produce the same result as simply using the best (maximum-likelihood) rule, even though averaging is performed over all possible rules in the space, the domains are highly noisy, and the samples are mediumto small-sized. In all cases, this is observed to be due to BMA's consistent tendency to assign highly asymmetric weights to di erent models, even when their accuracy di ers by little, with most models (often all but one) e ectively having no in uence on the outcome. Thus the e ective number of models being averaged is much smaller for BMA than for common ad hoc methods, leading to a smaller reduction in variance. This suggests that the success of the multiple models approach to rule induction is primarily due to this variance reduction, and not to its being a closer approximation to the Bayesian ideal.